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O (57) Abstract: An input device for interfacing with a computing device is provided. The input device includes a body configured to 
O be held within a human hand. The input device includes a light emitting diode (LED) affixed to the body and a power supply for the 
^ LED. A mode change activator is integrated into the body, where the mode change activator is configured to cause a change of a color 
Q of a light originating from the LED. The color change is capable of being detected to cause a mode change at the computing device. 
^ Methods for detecting input commands from an input source within a field of sight of an image capture device, and a computing 
system which includes the input device are provided. 
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Method and Apparatus for Light Input Device 

by Inventor: 
Richard L Marks 

5 Background of the Invention 

1. Field of the Invention 

[0001] This invention relates generally to alternative input mechanisms to computer systems, 
and more particularly to visually tracking a light, wherein a change in the light triggers an action 
on the part of the computer system. 

10 2. Description of the Related Art 

[0002] There has been a great deal of interest in searching for alternatives to input devices for 
computing systems. The keyboard and mouse for the desktop computing system are taken for 
granted at this time. However, for interactive entertainment applications in a "living room" 
environment, the keyboard and the mouse have failed to gain widespread acceptance. 

15 [0003] Tracking of moving objects using digital video cameras and processing the video images 
for producing various displays has been attempted also. However, these systems tend to rely on 
having a plurality of video cameras available for developing positional information about the 
object based on triangulation. The cost for these systems becomes prohibitive for their 
introduction into the "living room" environment 

20 [0004] However, in spite of the above knowledge and techniques, problems continue to hinder 
successful object tracking, and a particularly difficult problem is extracting precisely only those 
pixels of a video image which correspond unambiguously to an object of interest. For example, 
although movement of an object having one color against a solid background of another color, 
where the object and background colors vary distinctly from one another, can be accomplished 

25 with relative ease, tracking of objects, even if brightly colored, is not so easy in the case of 
multi-colored or non-static backgrounds. Changes in lighting also dramatically affect the 
apparent color of the object as seen by the video camera, and thus object tracking methods 
which rely on detecting a particular colored object are highly susceptible to error or require 
constant re-calibration as lighting conditions change. The typical home use environment for 

30 video game programs demands much greater flexibility and robustness than possible with 
conventional object tracking computer vision systems. 
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[0005] Thus, an alternative input device must be able to be tracked under the home use 
environment by a single relatively inexpensive camera in order to become widely accepted. 
Additionally, the alternative input device must be convenient to use. While a glove worn on the 
hand of a user, where the glove includes sensors that are tracked by a camera to capture input, 

5 has been trialed, users have not embraced the glove. One of the reasons for the lack of 
enthusiasm for a glove is the inconvenience of having to continually remove and put on the 
glove. Furthermore, the alternative input devices described above do not easily adapt 
themselves to being able to indicate a mode change, functionally similar to a mouse click. 
[0006] Thus, there is a need to solve the problems of the prior art to provide an input device 

10 capable of being tracked by a single video camera, wherein the input device is convenient for 
the user and capable of indicating a mode change. 

Summary of the Invention 

[0007] Broadly speaking, the present invention fills these needs by providing a method and 
system that provides a device capable of producing one or more colors of light (or variations of 

15 a single or multiple light shades) where a change or variation triggers a mode change to the 
system controlling a display of image data. As used herein, a mode change refers to a discrete 
event or action triggered through the detection of a change in light emitted from an input device. 
It should be appreciated that the present invention can be implemented in numerous ways, 
including as a process, a system, or a device. Several inventive embodiments of the present 

20 invention are described below. 

[0008] In one embodiment a method for triggering input commands of a program run on a 
computing system is provided. The method initiates with monitoring a field of view in front of 
an image capture device. Then, a light source within the field of view is identified. Next, a 
change in the light emitted from the light source is detected. In response to detecting the 

25 change, an input command is triggered at the program run on the computing system. 

[0009] In another embodiment, a method for detecting input commands from an input source 
within a field of sight of an image capture device is provided. The method initiates with 
minimizing an amount of light entering the image capture device. Then, a first color light 
signal is detected from the input source through the image capture device. Next, a change from 

30 the first color light signal to a second color light signal is detected. Then, a mode change is 
triggered in response to the change in the first color light signal. 
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[0010] In yet another embodiment, a computer readable medium having program instructions 
for triggering input commands of a program run on a computing system is provided. The 
computer readable medium includes program instructions for monitoring a field of view in front 
of an image capture device. Program instructions for identifying a light source within the field 
of view and program instructions for detecting a change in light emitted from the light source 
are included Program instructions for triggering an input command at the program run on the 
computing system in response to detecting the change are included 

[0011] In still yet another embodiment, a computer readable medium having program 
instructions for detecting input commands from an input source within a field of sight of an 
image capture device is provided. The computer readable medium includes program 
instructions for minimizing an amount of light entering the image capture device and program 
instructions for detecting a first color light signal from the input source through the image 
capture device. Program instructions for detecting a change from the first color light signal to a 
second color light signal and program instructions for triggering a mode change in response to 
the change in the first color light signal are provided. 

[0012] In another embodiment, a computing system having an input detection system, that 
determines when to trigger input commands of a main program run through the computing 
system is provided. The computing system includes an image capture device. Logic for 
monitoring a field of view associated with the image capture device and logic for tracking a 
position of a light source associated with an input object are included. Logic for detecting a 
color change in the light source and logic for triggering an input command at the main program 
run through the computing system, where the triggering is a result of the detected color change 
in the light source are included 

[0013] In yet another embodiment, an input device for interfacing with a computing device is 
provided. The input device includes a body configured to be held within a human hand. The 
input device includes a light emitting diode (LED) affixed to the body and a power supply for 
the LED . A mode change activator is integrated into the body, where the mode change activator 
is configured to cause a change of a color of a light originating from the LED. The color 
change is capable of being detected to cause a mode change at the computing device. 
[0014] Other aspects and advantages of the invention will become apparent from the following 
detailed description, taken in conjunction with the accompanying drawings, illustrating by way 
of example the principles of the invention. 



WO 2005/073838 



PCT/US2004/042865 



4 

Brief Description of the Drawings 

[0015] The invention, together with further advantages thereof, may best be understood by 
reference to the following description taken in conjunction with the accompanying drawings. 
[0016] Figure 1A is a simplified schematic diagram of a system having the capability of input 
detection from a light source in order to trigger a mode change in accordance with one 
embodiment of the invention. 

[0017] Figure IB is an alternative representation of the system depicted in Figure 1 A. 

[0018] Figure 2 is a simplified schematic diagram illustrating the capture of light from a light 

source through an image capture device in accordance with one embodiment of the invention. 

[0019] Figure 3 is a schematic diagram illustrating the determination of the location of a light 

source and the subsequent translation of that location to control movement of a corresponding 

cursor on a display screen in accordance with one embodiment of the invention. 

[0020] Figure 4 is a schematic diagram illustrating a scheme for enhancing a tracking and 

translation methodology in accordance with one embodiment of the invention. 

[0021] Figure 5 is a simplified schematic diagram illustrating a scheme for setting a scale 

dependent upon the distance a user is from the image capture device in accordance with one 

embodiment of the invention. 

[0022] Figure 6 represents alternative configurations of an input device in accordance with one 
embodiment of the invention. 

[0023] Figure 7 illustrates an alternative embodiment for the input devices illustrated in Figure 
6. 

[0024] Figure 8A is a simplified schematic diagram of a pair of input devices configured to 
communicate with a computing device in communication with a display monitor in accordance 
with one embodiment of the invention. 

[0025] Figure 8B is a schematic diagram of an alternative light input device configuration to the 
rings of Figure 7. 

[0026] Figure 9 is a flow chart diagram illustrating the method operations for triggering input 
commands for a program running on a computing system in accordance with one embodiment 
of the invention. 

[0027] Figure 10 is a flow chart diagram illustrating the method operations for detecting input 
commands from an input source within a field of sight of an image capture device in accordance 
with one embodiment of the invention. 
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Detailed Description of the Preferred Embodiments 

[0028] An invention is disclosed for an input device that is capable of emitting multiple 
colors/frequencies of light in order to trigger an event. In the following description, numerous 
specific details are set forth in order to provide a thorough understanding of the present 
5 invention. It will be apparent, however, to one skilled in the art that the present invention may 
be practiced without some or all of these specific details. In other instances, well known 
process steps have not been described in detail in order not to unnecessarily obscure the present 
invention. 

[0029] The embodiments of the present invention provide a user input device that is capable of 

10 emitting multiple colors of light which are captured through an image capture device. The 
changing from one light color to another initiates an event or action that can be displayed on a 
monitor. For example, the change from one light color to another can signify a mouse click, 
i.e., a mode change, to open a file or window for a personal computer, grab, drag or manipulate 
an image for a computer game, or even start a smart appliance. It should be appreciated that the 

15 input device includes some type of button, or actuator, which is used to change between light 
colors being emitted from a light emitting diode (LED). Thus, the input device will include 
some sort of power supply for the LED also. As will be explained by the embodiments 
described herein the user input device allows for the introduction of an experience that can be 
related to a mouse click while being convenient for the user. It should be appreciated that the 

20 light device is a "dumb" device. That is, no active communication takes place between the 
input device and the computer as the computer, in essence, watches for the light input device. 
In one embodiment, the light input device is configured as a flashlight. Of course, the light 
input device may be cordless, e.g., battery powered for the generation of light. 
[0030] Figure 1A is a simplified schematic diagram of a system having the capability of input 

25 detection from a light source in order to trigger a mode change in accordance with one 
embodiment of the invention. Image capture device 104 is in communication with computing 
device 106, which in turn is in communication with display screen 108. Input device 100 
includes a light source 102. For example, light source 102 may be a suitable light emitting 
diode (LED). light emanating from LED 102 is detected by image capture device 104. It 

30 should be appreciated that image capture device 104 may be any suitable image capture device 
capable of detecting light from LED 102. For example, any suitable web cam or other camera 
may be used as image capture device 104. The location in space of LED 102 is used to control 
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movement of cursor 110 on display screen 108 in one embodiment. That is, as input device 
100, and associated LED 102, is moved in space, that movement is translated in order to move 
cursor 110. Additionally, as will be explained in more detail below, input device 100, through 
LED 102, may cause a mode change, similar to the mode change initiated by clicking a mouse 
for a desktop computer. As mentioned above, a mode change refers to a discrete event or action 
triggered through the detection of a change in light emitted from an input device. In essence, 
input device 100 may be used as a computer mouse through color changes associated with LED 
102. Exemplary mode changes include mode changes associated with mouse-type operations 
where the movement features of the mouse in conjunction with the selection features, i.e., 
clicking, double clicking, and right clicking, are incorporated into the system described below. 
That is, the tracking features of the system capture the movement of the input device while the 
light color changes provide the means to capture selection features. While a mode change has 
been described in relation to mouse-type operations, it should be appreciated that the 
embodiments are not limited to mouse-type operations. That is, any suitable discrete operation 
or action is included here. 

[0031] Figure IB is an alternative representation of the system depicted in Figure 1A. Here, 
user 112 is holding input device 100 within his hand. LED 102 which is affixed to an end of 
input device 100, is within a detection region of image capture device 104. Image capture 
device 104 is in communication with computing device 106 which is in communication with 
display screen 108. In one embodiment, computing device 106 is any suitable video game 
console, e.g., the PLAYSTATION 2 console. Controller 116 is also in communication with 
computing device 106. Thus, user 112 may move input device 100 from an initial location, 
thereby changing the position of LED 102 relative to camera 104. This relative movement is 
then translated in order to move a cursor on display screen 108. Additionally, a mode change 
associated with the cursor being moved on display screen 108 is triggered through a change in 
the color of light emitted from LED 102. It should be appreciated that the embodiments 
described herein provide for a mouse-like device to be introduced in the living room for 
interactive entertainment and any other suitable application. 

[0032] Figure 2 is a simplified schematic diagram illustrating the capture of light from a light 
source through an image capture device in accordance with one embodiment of the invention. 
Here, user 112 is holding input device 100 which includes a light source thereon. Image 
capture device 104 monitors a field of view 118 through which light from the light source of 
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input device 100 is detected. The light source associated with input device 100 is within plane 
117, which corresponds to digitized screen 120. Here, an image of the light source associated 
with input device 100 is illustrated by region 122 of screen 120. It should be appreciated that 
the resolution of screen 120 may be associated with any suitable resolution typical of a web cam 
or other suitable camera. In one embodiment, screen 120 is defined by a screen size of 
320x240. Thus, as user 112 moves input device 100, the associated movement is captured 
through capture device 104 to determine a location of the light source within screen 120. It 
should be appreciated that the screen size and the imaging device size are decoupled. However, 
the screen and image device size are mapped in order to determine corresponding positions 
between the two. In one embodiment, the image device is mapped to a region of the screen. 
Heie, most of the screen is used for displaying a scene, game image, etc., and there is a 
relatively small input palette in a comer or some other suitable region of the screen. 
[0033] Figure 3 is a schematic diagram illustrating the determination of the location of a light 
source and the subsequent translation of that location to control movement of a corresponding 
cursor on a display screen in accordance with one embodiment of the invention. Here, screen 
120 defines an image of a light source as region 122. Region 122 includes portions of pixel P a , 
Pb, Pc, Pd, Pc, and P f . The remainder of each of the pixels in screen 120, i.e., all pixels except 
pixels Pa-Pf, are black. In one embodiment, ensuring that the remainder of the pixels is black is 
achieved through a masking operation. The masking operation includes reducing the size of an 
aperture of image capture device 104 in order to minimize an amount of light allowed into the 
image capture device. In one embodiment, the aperture size may be adjusted electronically by 
adjusting the sensor game and exposure time. This scheme enhances the ability to detect a light 
source while reducing interference effects associated with background lighting. It should be 
appreciated that since the characteristics of the light input device and the image capture device 
are known, then the image capture device parameters (white balance, gain, exposure, saturation, 
etc.) may be set explicitly to track a particular pre-determined pixel value, i.e., no calibration is 
required. As the input device is a light, the room lighting is not a factor here. Thus, an active 
method for detecting a light change is provided. 

[0034] Still referring to Figure 3, center 124 of region 122 is calculated through a centroid 
calculation in which the center's of each of pixels P a -Pf are calculated and then weighted 
according to the associated pixel value in order to determine the coordinates of center 124. The 
coordinates of center 124 are then mapped to display screen 128, which corresponds to the 
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display screen being viewed by the user. Thus, movement of the light source will cause 
movement of region 122 on grid 120, which may also be referred to as a screen associated with 
the image capture device. The corresponding movement of region 122 will be associated with 
the calculation of a new center. The new center will then be mapped to a location on screen 128 
in order to move cursor 130 on screen 128 so that the user is given a feeling of control, over the 
movement of cursor 130 through the LED input device. As will be explained in more detail 
below, the input device may have a button or some other suitable activation device which when 
pressed will cause the respective LED to change to a different color from a previous color. This 
different color is then captured by image capture device 104. The detection of the different 
color results in different pixel values being associated with the change in color. For example, 
the pixels corresponding to region 122 will be associated with different values by the color 
change. The different pixel values will then signal the mode change similar to a mode change 
signal associated with a mouse click. Thus, a user may click and drag, highlight, etc., images 
on the display screen. That is, the user may perform any functionality achieved through a 
mouse associated with a computer. 

[0035] In one embodiment, the centroid calculation is performed as described hereafter. The 
pixels not associated with pixels P a -Pf, are assigned a value of 0 as no light is detected, i.e., the 
pixels are black. It should be appreciated that the masking technique described above may be 
used to ensure that the image capture device can lock in on a light emanating from an input 
device by reducing interference from background lighting. Each of pixels P a -Pf are assigned a 
value corresponding to the amount of area of the pixel intersecting with region 122. In one 
embodiment, where pixel values are assigned from 0-255, 0 corresponding to no light, pixel P e 
is assigned the highest value while pixel P f is assigned the lowest value. For exemplary 
purposes the pixel values of pixels P a , P b , P c , Pd, P e , and P f are 121, 230, 80, 123, 240, and 10, 
respectively. Each of pixels P a -Pf is associated with a respective pixel center point Each of the 
two dimensional coordinates of each of the pixel centers is multiplied by the value of the 
respective pixel. These weighted values for one of the two dimensional coordinates are then 
summed together. In one embodiment, the summation of the weighted values for each 
coordinate is then divided by the summation of the pixel values associated with region 122 in 
order to provide the coordinates for the center of region 124. This technique may be described 
mathematically as: 
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(x,y)ccntcr= 2 [[(Xpixd ccatcr] [value of pixel for x pixfi i ccnter]]/£(pixel values) and 
2 [[(ypixei center] [value of pixel for y P ixeicenter]]/2(pixel values) 

[0036] Here, (x,y)center represent the two coordinates of center 124, xpixel center represents 
the x coordinate for each of pixels Pa -Pf , and ypixel center represents the y coordinate for each 
of pixels Pa -Pf. Thus, center 124 corresponds to a certain location of the image of the capture 
device. This position corresponds to a location on screen 128. With reference to video frames, 
(x,y)center may be calculated for each frame of the video and the location of (x,y)center is used 
to set a position of cursor 130 on screen 128. In one embodiment, a resolution associated with 
grid 120 is less than the resolution associated with screen 128, thereby enabling smooth 
movement of cursor 130 across screen 128. It will be apparent to one skilled in the art that a 
non-weighted centroid may also be determined, especially if the background is not known, e.g., 
the background is not all black. Here, the location of the centroid may not be as accurate as 
when the background is known, however, the accuracy is still suitable for the embodiments 
described herein. In one embodiment, the non-weighted centroid is calculated when the user is 
in a dark room or with an infrared LED and camera. It will be apparent to one skilled in the art 
that while Figures 1A, IB, 2, and 3 refer to a cursor, the embodiments are not limited to use 
with a cursor. In essence any suitable indicator that provides feedback on the second location of 
the input device may be used. For example, effects like distortion, brightening, darkening, 
telescope windowing, etc. may be employed to provide feedback on the second location of the 
input device. 

[0037] Figure 4 is a schematic diagram illustrating a scheme for enhancing a tracking and 
translation methodology in accordance with one embodiment of the invention. It should be 
appreciated that where a light source is captured through image capture device 104 and 
subsequently located within screen 120, the corresponding region associated with the light 
source is contained within one pixel. Thus, the subsequent translation to a cursor may cause the 
cursor movement to appear jumpy, due to the quantization effect of the discrete sampling of the 
image capture device. In order to alleviate the jumpiness, image capture device may be 
defocused to blossom or expand the region associated with the light source. For example, 
region 132 represents an initial capture of a corresponding light source. As can be seen, region 
132 is contained within one block of grid 120, which represents a single pixel. In order to 
expand or blossom region 132 the image capture device is defocused where regions 134 and 
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136 represent different defocusing parameters. Thereafter, the centroid of the expanded region 
may be calculated as discussed above. In one embodiment, a diffuser is placed over the LED to 
defocus the light source. For example, the diffuser may be a piece of tape that causes the light 
to diffuse. 

5 [0038] Figure 5 is a simplified schematic diagram illustrating a scheme for setting a scale 
dependent upon the distance a user is from the image capture device in accordance with one 
embodiment of the invention. Here, user 102a is at a first distance associated with image plane 
117a while user 102b is at a second distance corresponding to image plane 117b. It is not 
necessary that image capture device 104 has depth capability, as the corresponding scales from 

10 images captured at image plane 117a and image plane 117b may be used to provide a relative 
degree of distance corresponding to respective image areas occupied by user 102a and user 
102b. According to this relative degree of distance, the amount of movement for input device 
100 to cause a corresponding movement of a cursor on display screen 108 may be adjusted. For 
example, if the user is closer to image capture device 104, then larger movements may be used 

15 to correspond to a movement of a cursor as compared to smaller movements when the user is at 
a farther distance. 

[0039] Figure 6 represents alternative configurations of an input device in accordance with one 
embodiment of the invention. Input device 100a includes LED 102a-l and LED 102a-2 located 
at opposing ends of the input device. Button 103 is included on the body of input device so that 

20 a user may press the button in order to trigger a light change for a corresponding LED. More 
than one button may be incorporated into input device 100a in order to accommodate multiple 
LED's in one embodiment. Input device 100b includes a single LED 102b at an end of the 
input device. Here, LED 102b is capable of emanating multiple colors as button 103 is pressed. 
Input device 100c illustrates multiple LED's located adjacent to each other. Here, LED 102c- 1 

25 is adjacent to LED 102c-2. As an alternative to input device 100c, the input device may have a 
fork-type configuration where an LED is affixed to each of the ends of the prongs (tines) of the 
fork. Button 103 may also be used to trigger one of the LED's to emit light while another is off. 
Button 103 may also be referred to as a mode change activator. A mode change activator is 
broadly defined as any suitable mechanism that may be used to cause the LED to switch 

30 between colors of light and/or variations of colors of light. For example, the mode change 
activator may be a button, a switch, a rotary dial, etc. In addition, the LED's may be located 
elsewhere on the body of the input devices shown in Figure 6. For example, LED's may be 
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incorporated on the sides of the input device. Alternatively, a line of LED's along the side of 
the input device may be provided. In another embodiment, a large LED at an end of the input 
device is provided, thereby enabling a capture device to detect a change in shape when the input 
device is tilted- That is, the input device may be configured to enable the capture device to 
detect a change in angle of the input device relative to the capture device. For example, a user 
may angle the input device up, down or to the side in order to cause a certain mode change or 
response to the particular angle change. One skilled in the art will appreciate that numerous 
other suitable configurations are possible for the input device besides the configurations 
illustrated in Figure 6. Thus, the embodiments described herein are not limited to the 
exemplary configurations of Figure 6. 

[0040] Figure 7 illustrates an alternative embodiment for the input devices illustrated in Figure 
6. Here, input device 100 is configured as a remote control device which includes LED 102 and 
infrared capability indicated by light 140. Thus, the input device may be incorporated into a 
suitable remote control commonly used for television sets. In one embodiment, an LED 
capable of toggling between at least three colors is provided. Here, a third color may be used to 
provide functionality corresponding to a "right click" on a computer mouse. 
[0041] Figure 8A is a simplified schematic diagram of a pair of input devices configured to 
communicate with a computing device in communication with a display monitor in accordance 
with one embodiment of the invention. External input devices 100-1 and 100-2 are configured 
to fit over a finger or thumb of a user's hands 142a and 142b. As shown, each of input devices 
100-1 and 100-2 is capable of emanating light that is detected by image capture device 104 
which is in communication with computing device 106. While image capture device 104 is 
shown outside of the frame of computing device 106, it should be appreciated that the image 
capture device may be integrated into the computing device in one embodiment of the 
invention. Input devices 100-1 and 100-2 transmit different color light signals in one 
embodiment Computing device 106 is in communication with display monitor 108. 
Computing device 106 transmits digital data to display monitor 108 so that the digital data can 
be viewed. Display monitor 108 may display text 146a, menu 146b and/or graphics 146c. Of 
course, it should be noted that any suitable digital data may be displayed. In one embodiment, 
where computing device 106 is a game console, display monitor 108 displays graphics or a user 
interface associated with a game being played. 
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[0042] Each of input devices 100-1 and 100-2, of Figure 8A, is configured to generate different 
colors of light. In one embodiment, input devices 100-1 and 100-2 may emanate a common 
color of light and at least one different color of light It should be appreciated that button 144 
may be used to toggle between the different colors of light. The light may be emitted through a 
LED on the side of the ring band of input devices 100-1 and 100-2. Alternatively, input devices 
100-1 and 100-2 may be configured as thimbles, where a LED is affixed to the base of the 
thimble. In another embodiment, a single input device is employed rather than multiple 
devices. It should be appreciated that input devices 10O-1 and 100-2 are configured to fit 
comfortably over a user's finger or thumb. Thus, different sizes can be made available based 
upon a user's age, gender, etc. Alternatively, the input devices may be made to be adjustable. 
That is, elastic straps, or even VELCRO straps, may be incorporated to secure the input device 
to the user's finger or thumb in one embodiment of the invention. 

[0043] Figure 8B is a schematic diagram of an alternative light input device configuration to the 
rings of Figure 7. Here, thimble body 100-3 includes button 144 which is configured to change 
the light or frequency of light being emitted through LED 151. Of course, more than on LED 
may be located on the base of thimble 100-3, as discussed with reference to Figure 6. 
[0044] One skilled in the art will appreciate that the image capture device and the computing 
device include logic capable of providing the functionality described herein. The logic may 
include software elements and/or hardware elements. For example, the software elements may 
include software code processed through a central processing unit The hardware elements 
include logic gates and circuitry configured to achieve the functionality described herein. It will 
be apparent to one skilled in the art that the hardware elements, e.g., logic gates, may be 
synthesized to execute the functionality described herein. Of course, the system may include a 
combination of software and hardware elements interacting to provide the desired outcome also. 
[0045] Figure 9 is a flow chart diagram illustrating the method operations for triggering input 
commands for a program running on a computing system in accordance with one embodiment 
of the invention. The method initiates with operation 150 where a field of view in front of an 
image capture device is monitored. Here, the field of view may capture a light source 
emanating from an input device having an LED capable of emitting multiple colors of light, as 
described with reference to Figures 1 A through 2. The method then advances to operation 152 
where a light source within the field of view is identified. As described above, the light source 
emanates from the LED. The method then proceeds to operation 154 where a change in light 
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emitted from the light source is detected Here, a button may be pressed in order to change the 
color of light emitted from an LED which is then captured by the image capture device. That is, 
pixel values will change when the change in the color of light occurs. As used herein, the 
change in the color of light may refer to an actual color change, e.g. from red to green. 

5 However, the change of color may also refer to a variation of the color, e.g., one yellow variant 
to another yellow variant. In essence, any suitable change that will cause a corresponding 
change in the pixel values associated with the colors or variants may be used here. The method 
then moves to operation 156 where an input command at the program run on the computing 
system is triggered in response to the change in the color of light. For example, a mode change 

10 associated with a mouse click on a desktop computer may be triggered here. Thus, click and 
drag functionality, highlighting functionality, and any other suitable functionality achieved 
through a mouse click is capable of being introduced in the "living room" environment without 
the traditional mouse and keyboard hardware. In addition, the input command triggered by the 
light change can be an action, a movement cue, a modality change, etc. 

15 [0046] Figure 10 is a flow chart diagram illustrating the method operations for detecting input 
commands from an input source within a field of sight of an image capture device in accordance 
with one embodiment of the invention. The method initiates with operation 160 where an 
amount of light entering the image capture device is minimized. That is, an aperture of the 
image capture device may be reduced in order to mask background lighting. Thus, the 

20 minimization of the background lighting will enhance the tracking capability of a light source 
such as the input device described herein. For example, the reduction in the amount of light 
entering the image capture device may be achieved as described with reference to Figure 3. The 
method then advances to operation 162 where a first color light from the input source is 
detected through the image capture device as described with reference to Figures 2-4. Here, the 

25 first color light is detected and correlated to an image screen. 

[0047] The method of Figure 10 then proceeds to operation 164 where the first color light being 
changed to a second color light is detected. Here, the change in the color of light may be 
triggered through pressure applied to a button or mode change activator on the input device as 
described above. One skilled in the art will appreciate that the color change, or frequency 

30 change, may be detected by examining corresponding pixel values associated with an image 
capture device. The method then moves to operation 166 where a mode change is presented in 
response to the change in the first color light. The mode change may act to enable a user to 
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perform click-and-drag operations, highlighting operations, or any other suitable operations 
associated with a mode change such as enabled through a computer mouse. Additionally, when 
the second color light is changed back to the first color light, the end of a click-and-drag 
operation or highlighting operation is signified. Alternatively, a third color light may be used to 
enable functionality associated with a "right" mouse click. It should be noted that the mode 
change is displayed on a display screen similar to the mode change for a computer mouse in one 
embodiment of the invention. Additionally, one skilled in the art will appreciate that while 
Figure 10 is described with respect to color changes, the invention is not limited to color 
changes as variations of the same color of light may be used, e.g., shades of colors. 
Furthermore, different frequencies of light may be used instead of different colors. For 
example, infrared light may be used with a visible wavelength of light. As discussed above, any 
change to a light source that is capable of causing a change in pixel values associated with an 
image capture device may be used for the embodiments described herein. 
[0048] In summary, an input device capable of triggering a mode change through a multi-color, 
or multi-frequency, light generator is provided. Of course, multiple collocated lights having 
different colors or frequencies may be used to provided the same effect. Thus, any light 
generating structure can be used, whether it is a solid-state device such as a diode, or one or 
more conventional light bulbs. In one embodiment, the light is provided by one or more LEDs 
that can be coupled or installed onto a pen-like object. The tight emitted from the input device 
is tracked by an image capture device. The image capture device can take on any number of 
forms, including a camera, an array of charged coupled devices (CCDs), a digital camera, a 
conventional camera that is coupled to a digitizer, or a webcam In general, the image capture 
device should be able to detect light from the input device (or any object capable of emitting 
light). Preferably, the input device will generate enough light so as to enable the camera to 
detect a spot of light, which may be bright relative to other surrounding light. 
[0049] Thus, in one embodiment, similar to a computer mouse input device, the light from the 
input device may be set to always be on when it is detected by the camera. Thereafter, to cause 
control, the user may press a button, or some other suitable triggering mechanism, on the input 
device to change a color of light (or a frequency) being emitted. The webcam captures the 
color/frequency change, thereby enabling the color change to act as a button press for a mouse, 
i.e., mouse click. It should be noted that any suitable number of colors may be tracked and that 
the input device may have multiple buttons for the multiple colors or the input device may have 
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a single button that cycles through the multiple colors. In another embodiment, instead of 
changing visible color, the change may be from a first color that changes to general light, as is 
typical of standard flash lights. In still another embodiment, the input device can flicker 
between a first light and then to an off state, where not light is emitted at all. For example, an 

5 infrared LED may alternate between a first light and an off state. In such a case, when the light 
is off, the light may stay off for a set period of time. The timing of the off state can then cause 
one or more same, similar or different states or changes to occur on a display screen. 
[0050] Furthermore, the input device described herein enables enhanced functionality for 
interactive entertainment applications. For example, with respect to sports video games, a user 

10 may use the input device to design a play, e.g., a play for a football game, basketball game, etc. 
Strategy games, such as chess and war games, which were previously limited due to the 
difficulty in adopting the joystick controller to function as a mouse-type device become more 
user friendly when the input device described herein is used for mouse-type input, instead of the 
joystick. With respect to a video game that incorporates strategy, a user may be on a hillside 

15 directing armies to attack an enemy. In another embodiment, the input device may be used for 
television programming. The ability to incorporate a mode change associated with the change 
of a light color in conjunction with the capability to track the movement of the light source 
enables the accomplishment of these features. 

[0051] The invention may employ various computer-implemented operations involving data 
20 stored in computer systems. These operations are those requiring physical manipulation of 
physical quantities. Usually, though not necessarily, these quantities take the form of electrical 
or magnetic signals capable of being stored, transferred, combined, compared, and otherwise 
manipulated. Further, the manipulations performed are often referred to in terms, such as 
producing, identifying, determining, or comparing. 
25 [0052] Any of the operations described herein that form part of the invention are useful 
machine operations. The invention also relates to a device or an apparatus for performing these 
operations. The apparatus may be specially constructed for the required purposes, or it may be 
a general purpose computer selectively activated or configured by a computer program stored in 
the computer. In particular, various general purpose machines may be used with computer 
30 programs written in accordance with the teachings herein, or it may be more convenient to 
construct a more specialized apparatus to perform the required operations. 
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[0053] Although the foregoing invention has been described in some detail for purposes of 
clarity of understanding, it will be apparent that certain changes and modifications may be 
practiced. For example, although specific examples have been provided for use in relation to 
video gaming, the applications can be applied to any computer or computing device that will 
5 require some interaction. The computing device can be a single stand alone unit or can be 
interconnected to other computing devices over a local or global network, such as the Internet. 
Accordingly, the present embodiments aie to be considered as illustrative and not restrictive, 
and the invention is not to be limited to the details given herein, but may be modified within the 
scope and equivalents of the description. 
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What is claimed is: 
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Claims 

1. A method for triggering input commands of a program run on a computing 
system, comprising: 

monitoring a field of view in front of an image capture device; 
identifying a light source within the field of view; 
detecting a change in light emitted from the light source; and 
in response to detecting the change, triggering an input command at the program run 
on the computing system. 

2. The method of claim 1, wherein the change is one of a color change, and a 
light variation change. 

3. The method of claim 1, wherein the light source is a light emitting diode 
(LED) capable of emitting multiple colors of light. 

4. The method of claim 1, wherein the method operation of identifying a light 
source within the field of view includes, 

masking background light effects within the field of view. 

5. The method of claim 4, wherein the method operation of masking background 
light effects within the field of view includes, 

reducing an amount of light allowed into an aperture of the image capture device. 

6. The method of claim 1, wherein the input command causes a mode change 
linked to a cursor displayed on a display screen associated with the computing system. 

7. The method of claim 1, wherein the method operation of identifying a light 
source within the field of view includes, 

defining an area representing the light source within a grid associated with the image 
capture device; and 

expanding the area representing the light source within the grid. 
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8. The method of claim 7, wherein the method operation of expanding the area 
representing the light source within the grid includes, 

defocusing the image capture device relative to the light source, 

9. The method of claim 1, wherein the method operation of identifying a light 
source within the field of view includes, 

calculating a centroid of an image representing the light source through a grid 
associated with the image capture device. 

10. The method of claim 9, further comprising: 

translating coordinates of the centroid to a location on a display screen associated 
with the computing system; 

detecting movement of the light source within the field of view; and 

correlating the movement of the light source to movement of a cursor on the display 

screen. 

• 

11. A method for detecting input commands from an input source within a field of sight 
of an image capture device, comprising: 

minimizing an amount of light entering the image capture device; 

detecting a first color light from the input source through the image capture device; 

detecting a change from the first color light to a second color light; and 

presenting a mode change in response to the change in the first color light signal. 

12. The method of claim 11 wherein the method operation of minimizing an 
amount of light entering an image capture device includes, 

reducing an aperture size of the image capture device to enhance a signal representing 
light from the input source relative to other captured image data. 

13. The method of claim 12 wherein the method operation of reducing an aperture 
size of the image capture device results in filtering background light capable of interfering 
with light received from the input device. 



WO 2005/073838 PCT/US2004/042865 

19 

14. The method of claim 11 wherein the method operation of detecting a first 
color light signal from the light source through the image capture device includes, 

determining a location of a center of the first color light signal on a coordinate system 
associated with the image capture device; and 

mapping the location to a corresponding location on a display screen. 

15. Hie method of claim 11 wherein the method operation of detecting a change 
from the first color light to a second color light includes, 

detecting the second color light from the input source; and 

comparing pixel values associated with the first color light to pixel values associated 
with the second color light 

16. The method of claim 1 1 further comprising: 
reverting to the first color light from the second color light; and 

in response to reverting to the first color light, terminating the mode change. 

17. The method of claim 11 wherein the mode change is associated with one of a 
click and drag operation and a highlighting operation. 

18. The method of claim 1 1 wherein the first color light and the second color light 
originate from one of a single light emitting diode and multiple light emitting diodes. 

19. A computer readable medium having program instructions for triggering input 
commands of a program run on a computing system, comprising: 

program instructions for monitoring a field of view in front of an image capture 

device; 

program instructions for identifying a light source within the field of view; 
program instructions for detecting a change in light emitted from the light source; and 
program instructions for triggering an input command at the program run on the 
computing system in response to detecting the change. 
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20. The computer readable medium of claim 19, wherein the change is one of a 
color change and a light variation change. 

21. The computer readable medium of claim 19, wherein the light source is a light 
emitting diode capable of emitting multiple colors of light. 

22. The computer readable medium of claim 19, wherein the program instructions 
for identifying a light source within the field of view includes, 

program instructions for masking background light effects within the field of view. 

23. The computer readable medium of claim 22, wherein the program instructions 
for masking background light effects within the field of view includes, 

program instructions for reducing an amount of light allowed into an aperture of the 
image capture device. 

24. The computer readable medium of claim 19, wherein the input command 
causes a mode change linked to a cursor displayed on a display screen associated with the 
computing system. 

25. Hie computer readable medium of claim 19, wherein the program instructions 
for identifying a light source within the field of view includes, 

program instructions for defining an area representing the light source within a grid 
associated with the image capture device; and 

program instructions for expanding the area representing the light source within the 

grid. 

26. The computer readable medium of claim 25, wherein the program instructions 
for expanding the area representing the light source within the grid includes, 

program instructions for defocusing the image capture device relative to the light 

source. 
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27. The computer readable medium of claim 19, wherein the program instructions 
for identifying a light source within the field of view includes, 

program instructions for calculating a centroid of an image representing the light 
source through a grid associated with the image capture device. 

28. The computer readable medium of claim 27, further comprising: 

program instructions for translating coordinates of the centroid to a location on a 
display screen associated with the computing system; 

program instructions for detecting movement of the light source within the field of 
view; and 

program instructions for correlating the movement of the light source to movement of 
a cursor on the display screen. 

29. A computer readable medium having program instructions for detecting input 
commands from an input source within a field of sight of an image capture device, 
comprising: 

program instructions for minimizing an amount of light entering the image capture 

device; 

program instructions for detecting a first color light from the input source through the 
image capture device; 

program instructions for detecting a change from the first color light to a second color 
light; and 

program instructions for triggering a mode change in response to the change in the 
first color light signal. 

30. The computer readable medium of claim 29 wherein the program instructions 
for minimizing an amount of light entering an image capture device includes, 

program instructions for reducing an aperture size of the image capture device. 

31. The computer readable medium of claim 29 wherein the program instructions 
for detecting a first color light from the light source through the image capture device 
includes, 
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program instructions for determining a location of a center of the first color light on a 
coordinate system associated with the image capture device; and 

program instructions for mapping the location to a corresponding location on a 
display screen. 

32. The computer readable medium of claim 29 wherein the program instructions 
for detecting a first color light from the light source through the image capture device 
includes, 

program instructions for defocusing the image capture device with respect to the light 

source. 

33. The computer readable medium of claim 29 further comprising: 

program instructions for reverting to the first color light from the second color light; 

and 

program instructions for terminating the mode change in response to reverting to the 
first color light. 

34. A computing system, comprising: 
an image capture device; 

logic for monitoring a field of view associated with the image capture device; 
logic for tracking a position of a light source associated with an input object; 
logic for detecting a color change in the light source; and 

logic for triggering a mode change command at a main program run through the 
computing system, in response to the detected color change in the light source. 

35. The computing system of claim 34, wherein the computing system is one of a 
game console, a general computer, networked computer, and a distributed processing 
computer. 



36. The computing system of claim 34, wherein the logic for detecting a color 
change in the light source includes, 

logic for detecting a change in a pixel value associated with the light source; and 
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logic for detecting a change in a position of the light source relative to the image 
capture device. 

37. The computing system of claim 34, wherein each logic element is one or a 
combination of hardware and software. 

38. The computing system of claim 36, wherein the logic for detecting a change 
in a position of the light source relative to the image capture device includes, 

logic for calculating a centroid of an image representing the light source through a 
grid associated with the image capture device. 

logic for translating coordinates of the centroid to a location on a display screen 
associated with the computing system; 

logic for detecting movement of the light source within the field of view; and 

logic for correlating the movement of the light source to movement of a cursor on the 
display screen. 

39. The computing system of claim 38, wherein the logic for correlating the 
movement of the light source to movement of a cursor on the display screen includes, 

logic for adjusting a scale associated with translation of the movement of the light 
source to the movement of the cursor according to a distance of a user relative to the image 
capture device. 

40. The computing system of claim 34, further comprising: 

logic for minimizing an amount of light entering the image capture device in order to 
mask background light not associated with the light source. 

41. An input device for interfacing with a computing device, comprising: 
a body; 

a light emitting diode (LED) affixed to the body; 
a power supply for the LED; 
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a mode change activator integrated into the body, the mode change activator 
configured to cause a variation of a light originating from the LED , wherein the variation is 
capable of being detected to cause a mode change at the computing device. 

42. The input device of claim 41, wherein the mode change activator is 
configured to cycle between two variations of the light. 

43. The input device of claim 41, wherein the body includes a first end and a 
second end, a first LED affixed to the first end, a second LED affixed to the second end. 

44. The input device of claim 43, wherein the first LED emits a first variant of 
light and the second LED emits a second variant of light. 

45. The input device of claim 41 , further comprising: 
an infrared emitter. 

46. The input device of claim 41, wherein the body includes a first LED 
adjacently located to a second LED, the first LED emitting a first variant of light and the 
second LED emitting a second variant of light. 

47. The input device of claim 41, wherein the body is configured to be held 
within a human hand. 

48. The input device of claim 41, wherein the body is a ring configured to fit over 
a human finger. 

49. The input device of claim 41, wherein the body is thimble shaped and the 
LED is affixed to a base of the thimble shaped body. 

50. The input device of claim 41, wherein the mode change activator is 
configured to cycle between at least three light variant changes. 
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Fig. 2 



WO 2005/073838 



4/12 



PCT/US2004/042865 
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Fig. 5 
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Fig. 6 
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Fig. 7 
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